(19) 



J 



i 



(12) 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets (11) 

EUROPEAN PATENT APPLICATION 



EP 1 039 365 A2 



(43) 


Date of publication: 


(51) Int. CI/: OiUor o/Ooo 




27.09.2000 Bulletin 2000/39 




(21) 


Application nuiTiDer. uuotizo^o.** 






nafo of filinn* 99 H** 9finn 




(84) 


Designated Contracting States: 


(72) Inventor: Vuori, Petri 




AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


24100 Salo(R) 




MC NL PT SE 






Designated Extension States: 


(74) Representative: Haws, Helen et al 




AL LT LV MK RO SI 


Nokia I PR Department, 






Nokia House, 


(30) 


Priority: 26.03.1999 Fl 990676 


Summit Avenue 




Farnborough, Hampshire GU14 0NG (GB) 


(71) 


Applicant: 






NOKIA MOBILE PHONES LTD. 






02150 Espoo (Fl) 






(54) An input arrangement for manual entry of data and a mobile phone 

(57) The objective of the invention is a completely 
new input arrangement for manual entry of data to an 5 ^ 
electronic device (5) that comprises means to detect a 
temporary appearance of an obstacle in an input area 
(7A) outside the main body of the device (5) to deter- 
mine data input. Further the arrangement comprises 
means to project a picture on an input area (7A) on a 
surface (8). A virtual keyboard is projected on this sur- 
face (8) in front of or on a folding part of the apparatus 
in concern. A laser diode and diffractive optics is used to 
generate an image of the virtual keyboard on the sur- 
face (8) and infrared transmitters and receivers (9) are 
used to detect a pointer or a finger on an exact area of 
the virtual keyboard. The exact area or position is used 
to find out the key A - 0. ENT meant to be pushed. The 
virtual keyboard is preferably drawn to consist of bright 
red horizontal and vertical lines as well as letters, num- 
bers and possible special characters as "+" and The 
lines are bordering the areas of the individual keys. The 
invention is particularly suitable for a mobile phone. 
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Description 

[0001] The present invention relates to an input arrangement for manual entry of data forming a keyboard and/or a 
graphics input pad for a small electronic device and a mobile phone where this arrangement is applied to. 
5 [0002] Small mobile phone keyboards with keys 1 - 9, *, 0 and # as well as keyboards of so called communicators 
with alphanumerical keys reaching from a to z and 1 to 0 are known. These keyboards are small which causes difficulty 
to use. 

[0003] Also soft keyboards for electrical equipment are known. These keyboards are usually arranged as a touch- 
screen on a display. 

w [0004] As a problem in the known equipment is the space needed for the keyboard in a small apparatus. This forces 
to design the keyboard with few or little keys. A known device is represented in Figure 1 . A first part 1 A is seen contain- 
ing a display 2 and a second part 1 B a keyboard 3. The device 1 A, 1 B is kept for example on a surface 4 of a table. 
[0005] The present invention aims to provide a keyboard arrangement that enables a reasonable number of appro- 
priately sized keys to be realized even in a small-sized apparatus. 

15 [0006] The invention concerns an input arrangement for manual entry of data to an electronic device housed within 
a main body. According to one aspect of the invention the arrangement comprises means to detect a temporary appear- 
ance of an obstacle in an input area outside the main body of the device to determine input. 

[0007] The invention concerns also an electronic device. According to a further aspect of the invention the elec- 
tronic device comprises an input arrangement for manual entry of data with means to project a picture on an input area 

20 on a surface and means to detect the appearance of an obstacle on the surface in the input area to determine input. 
[0008] in consequence of the invention an input area is established in the close vicinity of the apparatus in concern. 
Advantageously a laser diode and diffractive optics is used to generate an image of a virtual keyboard on a surface and 
wireless, e.g. infrared transmitters and receivers are used to detect a pointer or a finger on an exact area of the virtual 
keyboard. The surface is usually a table-top. The exact area or position is used to find out the key meant to be pushed. 

25 The virtual keyboard is preferably drawn to consist of bright horizontal and vertical lines as well as letters, numbers and 
possible special characters as and "- M . The lines are bordering the areas of the individual keys. The optical technique 
for projecting the virtual keyboard is well known. 

[0009] The position of a pointer within the input area is preferably detected in a first direction extending preferably 
vertically out from the device from the attenuation of the infrared light and in a second direction parallel to the device 

30 from the place of the maximum reflection. The grid made of horizontal and vertical lines guides the user of the device 
to point in an area of a key. When the pointer crosses an infrared beam of light, it causes reflection that is detected in 
the device. The infrared beam has to be made very directional to enable detection by reflection of a beam. The detec- 
tion can be done also by choosing the direction of which the reflection is most effective. Crosstalk of detection between 
neighbouring directions can be decreased by using different wavelengths in adjacent transmitter receiver pairs. The 

35 detection from the attenuation of the infrared light is done using square-attenuation-law t/r 2 . The reflection voltage is 
then divided into ranges that are associated to discrete states. These ranges equal rows in the virtual keyboard. 
[0010] An advantage of the invention is that the device can be made small but a large virtual keyboard can be 
arranged. It is easy to touch keys that are large enough and all the alphabets can be arranged to the keyboard with dis- 
crete keys instead of number keys interconnected to different letters as in a number keypad. 

40 [001 1 ] Further advantages of the invention are reduction of cost and weight of the device. 
[0012] Preferable embodiments of the invention are represented in the dependent claims. 
[0013] The invention is described in detail in the following by referring to the attached drawings, where 



45 



50 



Figure 1 represents a known device with a keyboard, 

Figure 2 represents an arrangement with a virtual keyboard and an arrangement with a graphical input pad accord- 
ing to embodiments of the invention, 

Figure 3 represents an example of measured IR light reflection intensity, 

Figure 4 represents IR light reflection intensity according to an embodiment of the invention, 

Figure 5 represents some parts of a device according to the invention in a block diagram. 

55 Figure 6 represents some alternative arrangements for detecting the place of the pointer in the input area, and 

Figure 7 represents an arrangement for number key input to an embodiment of the invention in a mobile phone. 



2 



BNSOOCID: <EP 1039365A2J_> 




EP 1 039 365 A2 




[0014] Figure 1 is explained in the prior art section above. 

[001 5] In the Figure 2A an arrangement of a device 5 with a display 6 and a virtual keyboard 7A according to an 
embodiment of the invention is seen. The device 5 is situated on a surface 8 of a table. The device 5 and the virtual 
keyboard 7A is activated. The virtual keyboard 7A consisting of a text grid is projected from the device 5 using laser light 
5 ray 9A to the surface 8. The represented layout of the keyboard 7A is only exemplary and a QWERTY-type layout or any 
other meaningful layout can be arranged. 

[0016] In the Figure 2B is a more specific representation of the arrangement. The device 5 with the display 6 and 
the virtual keyboard 7 according to the invention and the surface 8 are present. The device 5 comprises infrared light 
transceivers 9 for all columns of the keyboard. Narrow infrared light beams are transmitted 9A over the columns and 

w received 9B if light is reflected from an obstacle 10 such as a finger or a pointer such as a pen back to the device 5. 
[0017] In the Figure 2C an arrangement of the device 5 with a display 6 and a virtual drawing plane or graphical 
input pad 7B according to an embodiment of the invention is seen. The device 5 is situated on a surface 8 of a table. 
The device 5 and the graphical input pad 7B is activated. The graphical input pad 7B consisting of borders and possibly 
function keys is projected from the device 5 using laser light ray to the surface 8. The position of the pointer 10 is 

75 detected exactly and a crossline for drawing is placed correspondingly on the display 6. Sketching a drawing is done by 
moving the pointer 10 in the input area 7B and looking at the trace on the display 6. It is also possible to implement a 
graphical input pad 7B without projecting any picture to the surface 8. 
[0018] In all Figures 2A, 2B, 2C the device 5 is shown from the front and the side. 

[001 9] In the Figure 3 is the attenuation described with an example of measured infrared light reflection intensity 
20 data from an input arrangement for manual entry of data corresponding to the invention. The infrared light intensity is 
detected using detectors that output a voltage corresponding to the intensity. The maximum seen output voltage of 
1700 mV is naturally reflected nearest to the device. The voltage corresponding to the reflection decreases rapidly 
below 600 mV that is only little above a noise floor of 400 mV. The measurement data has been acquired by moving a 
finger in straight line in the direction of the beam and the reflection is measured at 1 0 mm intervals. The noise floor 
25 detected with measurements is mainly caused by leakage flux from the transmitter to the receiver. At least the range 
from 10-70 mm is found to be applicable to the device by this measurement but also distances even to 100 mm could 
be used. The longest distance is here bound by the leakage flux. 

[0020] Measured reflection intensity of the IR light according to a prototype of the invention is shown in the Figure 
4 in two diagrams. 

30 [0021 ] Measured horizontal reflection curves are shown in the Figure 4A. The curves that are represented here are 
results of a measurement made by moving a testfinger through one column in the input area in a parallel direction to 
the side of the device at distances of 10, 30 and 70 mm. Measurement is done at 1 mm intervals. The results of the 
measurements are normalized to better illustrate proportional change in detecting the reflected light. In the normalized 
vertical scale the value 1 corresponds to the average of the result in the seen horizontal distance from the middle point. 

35 The horizontal movement of 1 mm is possible to be detected even at a distance of 70 mm with one detector. Anyway a 
more certain detection is achieved by using more than one detector at the same time to detect the position of the obsta- 
cle. The detectors can be differentiated by time divisional operation or wavelength. Actual detector voltage difference 
from a horizontal finger movement of 1 0 mm from the middle point is 782 mV at a distance of 1 0 mm, 128 mV at 30 mm, 
47 mV at 50 mm and 12 mV at 70 mm. 

40 [0022] Measured distance attenuation curve is shown in the Figure 4B. The represented curve is measured by 
moving a testfinger in a transversal direction to the side of the device to a distance of 70 mm. The result of the meas- 
urement is represented in actual value. Three regions of the curve are observed 0-5 mm, 6-27 mm and 28 - 70 mm. 
The first region A is affected by a printed circuit board shadowing the receiver, the second region B is dominated by 
direct reflection from the finger and the third region C is dominated by diffuse reflection from the finger. The diffuse 

45 reflection is caused by environmental infrared radiation reflecting from the finger. In addition to the above mentioned the 
finger itself radiates some infrared light. The amount of the reflection from the finger depends also on fingernails, skin 
and the size of the finger. A vertical movement of 1 mm is possible to be detected even at a distance of 70 mm. Detector 
voltage difference from a vertical finger movement of 1 mm is 24 mV at a distance of 10 mm, 9 mV at 30 mm, 4 mV at 
50 mm and 3 mV at 70 mm. 

so [0023] It is also possible to implement the key input arrangement of the Figure 2A using only half as many infrared 
light transceivers compared to what is shown in the Figure 2B. In the Figure 2B the number of the transceivers and key 
columns is equal. The following is also applicable to the drawing input represented in the Figure 2C. Realisation is pos- 
sible by simultaneous use of two concurrent transceivers to detect the place of the obstacle in front of the device even 
between lines perpendicular to the device extending outwards from the transceivers using reflection measurement data 

55 represented in the Figures 4A and 4B with calibration. This is based on the width of the measured horizontal reflection 
curves and the attenuation when the obstacle is moved from the centre line where the reflection is at its maximum. The 
distance of the obstacle from the device is calculated using a preformed table to correct the higher of the two strongest 
detected value. The table is formed using attenuation measurement data when calibrating the device. An exemplary 
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[0024] The left column of the table shows the absolute voltages 1 .0, 2.0, 3.0 of the detector with stronger reading. 
The top row shows the ratio V^/v^ where V 1f V 2 are the voltages of the detectors. The element K X( y contains a correc- 
ts - 'actor for correcting the stronger voltage to get a voltage proportional to direct distance from the obstacle to the 

[CtiZH] The horizontal placing of the obstacle is defined nearer to either one of the optical lines reaching vertically 
o - om the transceivers of the device using the ratio V 1 /V 2 . If the ratio is greater than 1 , the obstacle is nearer the opti- 
ca tsne of the detector 1 and if the ratio is less than 1 , the obstacle is nearer the optical line of the detector 2. If the ratio 
is 1 . the obstacle is halfways between the optical lines of the detectors 1 and 2. To find accurate horizontal placing suit- 
able compensation factors has to be used. These factors are arranged in a table for different distances from the device 
and for different deviations from the optical line of the detector. 

[0028] In the Figure 5 some important parts of a device according to an embodiment of the invention are seen in a 
biock diagram. The device comprises a processor 1 1 to run a program, a memory 12 to record the program, a display 
13 to show information, an input area picture projector 14 to control the drawing of a picture to an input area, a laser 
diode 15 to form light to draw the picture, an input detector unit 16 to detect a pointer in the input area, infrared diodes 
1 7 to form infrared light beams upon the input area and infrared sensors 1 8 to sense reflection of infrared light from the 
pointer. 

[0027] In the Figure 6 alternative arrangements for detecting the place of the pointer in the input area are seen. 
[0028] In the Figure 6A is a phase-difference detection arrangement. The device 19 comprises a display 20, infra- 
red light transmitter 21 and detectors 22 A, 22B, 22C. The position of the pointer 23 in the area of the virtual keyboard 
24 is detected by phase-differences in reception of reflected light from the pointer 23 to the detectors 22A, 22B, 22C 
along the side of the device 19 and the virtual keyboard 24. The distances from the pointer 23 to the detectors 22A, 
22B, 22C are calculated using the phase-differences. The distances are calculated using the following equations: 



A<p = 2*r'Af, 



0) 



40 



Af = Ar/c, (2) 
where Aq> is phase difference, f is frequency, At is time difference. Ar is distance difference and c is speed of light. 

A<pc 



Ar = 



2nf 



(3) 
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the difference of distance Ar from the pointer 23 to the detector 22A, 22B, 22C is calculated from the phase difference 
A<p. The actual position of the pointer 23 is calculated using distance differences Ar to at least three detectors 22A, 22B, 
22C. The possible distance differences Ar can be arranged in a table to easily find corresponding x, y coordinates or 
virtual keys. This arrangement is independent on transmitter characteristics neither is a uniform infrared field needed. 
[0029] The infrared light transmitter 21 and the detectors 22A. 22B, 22C can be replaced with acoustical ones in all 
embodiments of the invention. Sound propagation speed in air is approximately 340 m/s at room temperature. Propa- 
gation of 1 mm in 3 fis is detectable with detectors 22A, 22B, 22C. This enables a resolution of circa 1 mm in all places 
of the input area. 

[0030] In the Figure 6B is a polar-radiator detection arrangement. The device 25 comprises a display 26 and omni- 
directional transmitter/receivers 27A, 27B. The pointer 28 is in an input area or the area of the virtual keyboard 29. The 
transmitter/receivers 27A, 27B send infrared light the light reflects from the pointer 28 and is received preferably by the 
same transmitter/receiver 27A, 27B. The transmitter/receivers 27A, 27B can use different wavelength and/or time of a 
pulsed transmission to differentiate from each other. Attenuation of light is used to determine the distance of the pointer 
from the transmitter/receivers 27A, 27B. 
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[0031] A minimum number of two transceivers are needed for a polar detection arrangement, but the accuracy can 
be enhanced using three or more transceivers. 

[0032] In the Figure 6C is a shadow-based detection arrangement. The device 30 comprises display 31, infrared 
light transmitter 32 and a photodetector strip 33. The position of the pointer 34 is detected by measuring the place and 

5 size of the shadow 35 of the pointer 34 on the photodetector strip 33. The pointer 34 has to be of a known constant size 
to enable reliable detection of the distance between the transmitter 32, pointer 34 and the shadow 35. The shadow 35 
grows when the pointer 34 is moved towards the transmitter 32 and shrinks when the pointer 34 is moved towards the 
photodetector strip 33. Also the sharpness of the shadow 35 can be used to determine or confirm the result. The 
shadow 35 becomes blunt when the pointer 34 is moved towards the transmitter 32 and sharpens when the pointer 34 

10 is moved towards the photodetector strip 33. 

[0033] It is also possible to make a matrix-type detection arrangement where there are both intensity and shadow- 
based detection equipment in the same device. The different detection systems are used parallel to measure the verti- 
cal position with the first and the horizontal position with the second method. This structure is anyhow slightly more 
complicated and expensive to manufacture. 

75 [0034] In the Figure 6D is a centre radiator arrangement. The device 36 comprises a display 37, an infrared light 
transmitter 38 and a receiver 40. The transmitter 38 sends infrared light to all horizontal directions. The light reflects 
from an obstacle 39 to the receiver 40 of the device. 

[0035] In the Figure 6E is the receiver 40 of the arrangement of the Figure 6D represented. The reflected light from 
the obstacle 39 arrives in an angle 9 to the normal of the device. The receiver 40 comprises a row of detectors 41 and 
20 a set of photoconductors 42. Every individual photoconductor has a known angle of incidence compared to the normal. 
The incidence of the reflection is found out as a light detection in a detector of that angle. If the reflection is detected by 
many detectors the angle of incidence is found out by the strongest detection. The distance to the obstacle is calculated 
from the attenuation as above. 

[0036] In the Figure 7 is an arrangement for number key input according to an embodiment of the invention in a 
25 mobile phone. The mobile phone 43 comprises a display 44, number key markings 45A on the phone and transmitters 
and receivers 45B on the side of the phone. Transmitters emit infrared light rays 46A that reflect 46B back to the phone 
43 from an obstacle 47 if one is present. The phone can be kept on a surface 48 but it is not necessary using this 
embodiment of the invention. The key input is preferably arranged as in a known mobile phone allowing text input by 
sequential key strikes on a key. 
30 [0037] The following example is represented to explain the invention for inputting text. The word "ice" is written and 
entered using the virtual keyboard according to the invention. The user switches on the device 5, the display 6 and the 
virtual keyboard 7. The laser diode is used to form an illusion of a keyboard on a surface 8 in front of the device. The 
user put his finger on the place where letter "I" is projected and an infrared light beam is reflected back from the finger 
on the column of the keyboard where the letter is placed. The reflection is recognized in the device 5 and a letter "I" is 
35 input as well as drawn to the display 6. Then the user put his finger on letter "C" and then on letter "E" and the corre- 
sponding letters are input to the device 5 and shown on the display 6. Inputting the word is ended by putting finger on 
the picture of the key ENT (Enter). 

[0038] A virtual keyboard with 8 * 7 square keys of 1 0 mm side length has been found suitable. Pushes to the keys 
of this keyboard can be read reliably. . 
40 [0039] The input area can be at least in front of the device as well as on a side of the device. Further it is possible 
to form a round shaped device and arrange the input area round. One example of a round device is a watch with the 
arrangement of the invention on a circular bezel or the case. The input area can also be arranged on a special surface 
that is a folding part of the device. 

[0040] An obstacle in sense of this text can at least be a finger, a pen or a special pointing stick. 
45 [0041] Calibration before use is usually needed. This depends on the form and quality of the source infrared field. 
In strong light the transmission and reception of the infrared light has to be filtered to a specified wavelength to differ- 
entiate from the lighting of the environment 

[0042] Current consumption of a certain prototype of the virtual keyboard according to the invention is 1 - 2 mA 
RMS (Root-Mean-Square) in the above represented size. The size on the keyboard can however be enlarged by chang- 
50 ing alignment of the laser optics and fortifying the infrared light transmission. This increases current consumption. Infra- 
red emission is preferably pulsed to reduce current consumption. 

[0043] Windows for infrared light are not necessarily needed in the cover of the electronic device if cover plastics 
that let infrared light pass through are used. 

[0044] The terms column and row describing the virtual keyboard in this text are only exemplary. The structure of 
55 the keyboard can differ from the above shown. The direction of the keyboard compared to the device can also differ. 
The keyboard can be for example on a side of the device. 

[0045] A further target is sensing dynamically the lowering and the rising of the finger to enter lower case letters or 
capital letters by making shallower or deeper press. 
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[0046] it is also possible to arrange multiple input areas to cover different input methods. One possibility is to 
arrange a virtual drawing plane on the left, a virtual keyboard on the front and a virtual number keyboard on the right 
side of the device. 



5 Claims 

1. An input arrangement for manual entry ol data to an electronic device housed within a main body, characterized 
in that the input arrangement comprise means (9; 16, 17, 18) to detect a temporary appearance of an obstacle (10) 
in an input area (7 A, 7B) outside the main body (5) of the device to determine input. 

70 

2. An arrangement according to the Claim 1 , characterized in that the means (16) to detect placing of the obstacle 
(10) comprise an infrared transmitter to emit infrared radiation into the input area (7 A, 7B) and a receiver to meas- 
ure infrared radiation reflected from the obstacle. 

75 3. An arrangement according to the Claim 1, characterized in that the means (16) to detect placing of the obstacle 
(10) comprise means (11, 12, 13) to calibrate preprogrammed correlation data for a user by placing the obstacle 
(10) on a specified place in the input area (7A. 7B). 

4. An arrangement according to the Claim 1 , comprising means (14) to project a picture on the input area (7 A, 7B) 
20 on a surface (8). 

5. An arrangement according to the Claim 4, characterized in that the surface (8) is a flat external surface which the 
device (5) lies on. 

25 6. An arrangement according to the Claim 4, characterized in that the surface is on a folding part of the device. 

7. An arrangement according to the Claim 4, characterized in that the means (14) to project the picture on the input 
area comprise laser optics (15). 

30 8. An arrangement according to the Claim 4, characterized in that the projected picture on the input area (7 A) repre- 
sents a virtual keyboard comprising at least a row of virtual keys and means (16, 17, 18) to detect the temporary 
appearance of the obstacle (1 0) are arranged to determine a keypush of the virtual keyboard. 

9. An arrangement according to the Claim 8, characterized in that the means (16, 17, 18) to detect the temporary 
35 appearance of the obstacle (10) are arranged to determine the row of the key pushed by measuring the intensity 

of the reflection and preprogrammed correlation data for attenuation compared to the distance of the obstacle. 

10. An arrangement according to the Claim 8, characterized in that the means (16, 17, 18) to detect the temporary 
appearance of the obstacle (10) are arranged to determine the position of the key pushed in a row is determined 

40 by measuring the intensity of the reflection and using preprogrammed correlation data for attenuation compared to 
the distance of the obstacle. 

1 1. An arrangement according to the Claim 4, characterized in that means (14) to project a picture on the input area 
are arranged to project borders of a graphical input pad (7B) and means (16, 17, 18) to detect the temporary 

45 appearance of the obstacle (1 0) are arranged to determine graphical input on the graphical input pad (7B). 

12. An arrangement according to the Claim 11, characterized in that means (16, 17, 18) to detect the temporary 
appearance of the obstacle (10) comprise at least one directional infrared transmitter and receiver pair (9) for the 
input area to determine the horizontal place of the intended input by sensing reflection. 

50 

13. An arrangement according to the Claim 11. characterized in that means (16, 17, 18) to detect the temporary 
appearance of the obstacle (10) are arranged to determine the vertical place of the intended input by measuring 
the intensity of the reflection and preprogrammed correlation data for attenuation compared to the distance of the 
obstacle (10). 

55 

14. An arrangement according to the Claim 1, characterized in that means (16, 17, 18) to detect the temporary 
appearance of the obstacle (10) in an input area outside the main body of the device (5) to determine input are 
arranged to read number key entry and the input area consists of a plane extending out from the side of the device 
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(5). 

5. An electronic device, characterized in that the electronic device comprises an input arrangement for manual entry 
of data with means (14, 1 5) to project a picture on an input area on a surface and means (16, 1 7, 1 8) to detect the 
temporary appearance of the obstacle (10) on the surface in the input area to determine input. 

6. An electronic device according to the Claim 15, characterized in that the device (5) further comprises 

a processor (1 1) to run a program, 

a memory (12) to record the program, 

a display (1 3) to show information, and 
the means to project a picture comprise 

an input area picture projector (14) to control drawing of a picture to the input area, 

a laser diode (15) to form light to draw the picture, and 
the means to detect placing an obstacle on the surface comprise 

an input detector unit (16) to detect a pointer in the input area, 

infrared diodes (17) to form infrared light beams upon the input area, and 

infrared sensors (18) to sense reflection of infrared light from the pointer. 

7. An electronic device according to the Claim 15, characterized in that the electronic device (5) is a mobile phone. 
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